To deal with the crowd supervision issue of low accuracy and poor real-time performance in public places, an abnormal crowd behavior detection approach based on motion saliency map was proposed. Firstly, the Lucas-Kanade method was used to calculate the optical flow field of the sparse feature points, then the movement direction, velocity and acceleration of feature points were computed after filtering the optical flow field both in time and space. In order to precisely describe the crowd behavior, the amplitude of velocity, the direction change, and the amplitude of acceleration were mapped to three image channels corresponding to R, G, and B, respectively, and the motion saliency map for describing the characteristics of crowd movement was fused by this way. Finally, a convolution neural network model was designed and trained for the motion saliency map of crowd movement, and the trained model was used to detect abnormal crowd behaviors. The experimental results show that the proposed approach can effectively detect abnormal crowd behaviors in real time, and the detection rate in the datasets of UMN and PETS2009 are more than 97.9%.